In the existing block based video coding methods, there are basically two types of coding modes: Intra mode and Inter mode. Intra mode only uses the information within a current picture, while Inter mode can use the information in other pictures (forward or backward) as reference.
When encoded video is transmitted over error prone network, errors happened in one picture will propagate to the subsequent pictures when Inter mode is used due to the prediction of Inter mode, while coding in Intra mode can suppress error propagation very well since it does not refer to former data. Therefore, forced Intra coding, also called Intra refresh, is one of the most fundamental and useful tools for error resilient video coding. It may quickly recover video subjective quality. But encoding the whole picture in Intra mode (I-picture) produces a large amount of bits which will cause rate fluctuation. In the hybrid video coding framework many other Intra refresh methods have been proposed, which refresh only a part of a picture to be Intra mode by some special rules.
Most of the existing Intra refresh methods use random refresh, periodical refresh, motion information based refresh, Rate Distortion (RD) model based refresh, packet loss rate (PLR) based refresh, feedback based refresh, or a combination of these methods.
P. Haskell and D. Messerschmitt “Resynchronization of motion compensated video affected by ATM cell loss”, ICASSP-92, March 1992, proposed that blocks with most rapid changes should be Intra refreshed most often, using the periodical random Intra refresh and motion information based conditional Intra refresh. MPEG-4 adopted this idea as Adaptive Intra refresh (AIR), with calculating the sum of absolute difference (SAD) of co-located blocks in consecutive pictures to know whether the block is in the motion area. In G. Cote, F. Kossentini “Optimal intra coding of blocks for robust video communication over the Internet” Image Commun. September 1999, an end-to-end RD model based intra refresh was proposed, wherein the error concealment is considered and the estimated end-to-end distortion as well as assumed PLR is combined into the RD optimization criterion. Based on this, Y. Zhang, W. Gao, H. Sun et al, “Error Resilient video coding in H.264 Encoder with Potential Distortion Tracking,” IEEE International Conference on Image Processing, October 2004, described a simple and efficient RD model using the block-based distortion map to simulate the potential error propagation.
Other methods have also been proposed, such as isolated region based intra refresh in Y. Wang, M. Hannuksela, M. Gabbouj: “Error-robust inter/intra macroblock mode selection using isolated regions,” Proceedings of 13th Packet Video Workshop, April 2003, which introduces a gradually growing region named isolated region, and the prediction area for the blocks in the isolated region is restricted.
Different from the above intra refresh methods, the European patent application EP05300975 proposes an attention-based adaptive Intra refresh for error resilient video coding, which takes subjective human vision property into account, so that a much better subjective quality is achieved than with former Intra refresh methods under the same bit rate and packet loss conditions. The flowchart of attention-based Intra refresh is shown in FIG. 1. An attention area extractor 11 extracts attention area information 14 and provides it to an Intra refresh controller 12, which sets the blocks in the attention area to be encoded 13 with higher priority in intra mode.
However, one important problem not considered in all these former Intra refresh methods, which limited their usage in real application systems, is that there is no rate control scheme available especially for Intra refresh. On one side, all the Intra refresh schemes are originally designed for error resilience and are studied with constant quantization parameters (QP) which makes it hard to achieve an appointed bit rate and can not prevent buffer overflow and underflow. But in real application systems, a certain bit rate is required and rate control has to be developed to control the buffer fullness and to obtain a fluent video quality.
On the other side, because of the different properties of Intra coding and Inter coding, Intra refresh changes the rate distortion model a lot, which is a key factor for rate control. For example, Intra coding will generate much more bits than Inter coding under the same QPs. If the same bit amount is allocated to Intra coding as to Inter coding, the subjective quality of Intra coding will be heavily degraded. Therefore the common bit allocation method in rate control, which evenly allocates bits to each Inter picture, causes a drastic fluctuation in subjective quality when used with Intra refresh. Accordingly, a more appropriate rate control method is needed for Intra refresh in error resilient video coding.
Rate control for video coding can usually be classified into GOP (group of pictures) layer, picture layer (frame or field), slice layer and macroblock (MB) layer. In each case, the rate control framework mainly has the two steps of bit allocation, wherein a certain amount of bits is allocated to the processed unit as target bits according to some criteria, and QPs decision, wherein proper quantization parameters are computed by some schemes to encode the unit with the allocated target bits. Most of the studies on rate control were done on the second step, and various rate distortion models have been proposed to illuminate the relationship among rate, distortion and QP. Examples are the linear rate-distortion model in TM5 for MPEG-2, quadratic rate-distortion model in TMN8 for H.263 and VM8 for MPEG-4.
Also taking the attention model and subjective quality into account, the European patent application EP06300184 proposed a content-based distortion-driven bit allocation scheme, which provides a reasonable bit allocation method for different attention areas within a picture and achieves a much better subjective quality within each picture.
Although rate control has been long investigated for video coding, only traditional video sequence IPPP . . . or IBBPBBP . . . structures have been considered. However, when Intra refresh is introduced, many Intra coded blocks are inserted in the previously Inter coded picture (P- or B-picture), and the RD model changes due to the different properties of Intra coding and Inter coding. So the existing rate control methods do not work well for error resilient video coding with Intra refresh. Furthermore, the location and proportion of Intra refreshed blocks are usually variable and depend on video content and Intra refresh schemes. This increases the difficulty of rate control.
For these reasons, many effective Intra refresh methods are restricted and may not well be used in real applications.